{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Symbol Tutorial\n",
    "\n",
    "Besides the tensor computation interface [NDArray](./ndarray.ipynb), another main object in MXNet is the `Symbol` provided by `mxnet.symbol`, or `mxnet.sym` for short. A symbol represents a multi-output symbolic expression. They are composited by operators, such as simple matrix operations (e.g. “+”), or a neural network layer (e.g. convolution layer). An operator can take several input variables, produce more than one output variables, and have internal state variables. A variable can be either free, which we can bind with value later, or an output of another symbol. \n",
    "\n",
    "## Symbol Composition \n",
    "### Basic Operators\n",
    "The following example composites a simple expression `a+b`. We first create the placeholders `a` and `b` with names using `mx.sym.Variable`, and then construct the desired symbol by using the operator `+`. When the string name is not given during creating, MXNet will automatically generate a unique name for the symbol, which is the case for `c`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Symbol a>, <Symbol b>, <Symbol _plus0>)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import mxnet as mx\n",
    "a = mx.sym.Variable('a')\n",
    "b = mx.sym.Variable('b')\n",
    "c = a + b\n",
    "assert a.name == \"a\", \"Symbol name incorrect.\"\n",
    "assert b.name == \"b\", \"Symbol name incorrect.\"\n",
    "(a, b, c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most `NDArray` operators can be applied to `Symbol`, for example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# elemental wise times\n",
    "d = a * b  \n",
    "# matrix multiplication\n",
    "e = mx.sym.dot(a, b)   \n",
    "# reshape\n",
    "f = mx.sym.Reshape(a, shape=(2,6))  \n",
    "# broadcast\n",
    "g = mx.sym.broadcast_to(f, shape=(3,2,6))  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Neural Networks\n",
    "\n",
    "Besides the basic operators, `Symbol` has a rich set of neural network layers. The following codes construct a two layer fully connected neural work and then visualize the structure by given the input data shape. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.36.0 (20140111.2315)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"214pt\" height=\"506pt\"\n",
       " viewBox=\"0.00 0.00 214.00 506.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 502)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-502 210,-502 210,4 -4,4\"/>\n",
       "<!-- data -->\n",
       "<g id=\"node1\" class=\"node\"><title>data</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">data</text>\n",
       "</g>\n",
       "<!-- fc1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>fc1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-168 -7.10543e-15,-168 -7.10543e-15,-110 94,-110 94,-168\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-142.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-127.8\" font-family=\"Times,serif\" font-size=\"14.00\">128</text>\n",
       "</g>\n",
       "<!-- fc1&#45;&gt;data -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>fc1&#45;&gt;data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-99.8131C47,-86.1516 47,-71.0092 47,-58.3283\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-109.906 42.5001,-99.9062 47,-104.906 47.0001,-99.9062 47.0001,-99.9062 47.0001,-99.9062 47,-104.906 51.5001,-99.9062 47,-109.906 47,-109.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-80.3\" font-family=\"Times,serif\" font-size=\"14.00\">200</text>\n",
       "</g>\n",
       "<!-- relu1 -->\n",
       "<g id=\"node3\" class=\"node\"><title>relu1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"94,-278 -7.10543e-15,-278 -7.10543e-15,-220 94,-220 94,-278\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-252.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-237.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu1&#45;&gt;fc1 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>relu1&#45;&gt;fc1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-209.813C47,-196.152 47,-181.009 47,-168.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-219.906 42.5001,-209.906 47,-214.906 47.0001,-209.906 47.0001,-209.906 47.0001,-209.906 47,-214.906 51.5001,-209.906 47,-219.906 47,-219.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-190.3\" font-family=\"Times,serif\" font-size=\"14.00\">128</text>\n",
       "</g>\n",
       "<!-- fc2 -->\n",
       "<g id=\"node4\" class=\"node\"><title>fc2</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-388 -7.10543e-15,-388 -7.10543e-15,-330 94,-330 94,-388\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-362.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-347.8\" font-family=\"Times,serif\" font-size=\"14.00\">10</text>\n",
       "</g>\n",
       "<!-- fc2&#45;&gt;relu1 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>fc2&#45;&gt;relu1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-319.813C47,-306.152 47,-291.009 47,-278.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-329.906 42.5001,-319.906 47,-324.906 47.0001,-319.906 47.0001,-319.906 47.0001,-319.906 47,-324.906 51.5001,-319.906 47,-329.906 47,-329.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"57.5\" y=\"-300.3\" font-family=\"Times,serif\" font-size=\"14.00\">128</text>\n",
       "</g>\n",
       "<!-- out_label -->\n",
       "<g id=\"node5\" class=\"node\"><title>out_label</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"159\" cy=\"-359\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-355.3\" font-family=\"Times,serif\" font-size=\"14.00\">out_label</text>\n",
       "</g>\n",
       "<!-- out -->\n",
       "<g id=\"node6\" class=\"node\"><title>out</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"170,-498 76,-498 76,-440 170,-440 170,-498\"/>\n",
       "<text text-anchor=\"middle\" x=\"123\" y=\"-465.3\" font-family=\"Times,serif\" font-size=\"14.00\">SoftmaxOutput</text>\n",
       "</g>\n",
       "<!-- out&#45;&gt;fc2 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>out&#45;&gt;fc2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M97.2538,-431.413C87.3525,-417.343 76.2083,-401.506 66.9347,-388.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"103.23,-439.906 93.7952,-434.318 100.353,-435.817 97.4753,-431.728 97.4753,-431.728 97.4753,-431.728 100.353,-435.817 101.155,-429.138 103.23,-439.906 103.23,-439.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"97\" y=\"-410.3\" font-family=\"Times,serif\" font-size=\"14.00\">10</text>\n",
       "</g>\n",
       "<!-- out&#45;&gt;out_label -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>out&#45;&gt;out_label</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M135.648,-430.057C140.313,-416.062 145.505,-400.485 149.807,-387.579\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"132.365,-439.906 131.258,-428.996 133.946,-435.163 135.527,-430.419 135.527,-430.419 135.527,-430.419 133.946,-435.163 139.796,-431.842 132.365,-439.906 132.365,-439.906\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f55ac034950>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Output may vary\n",
    "net = mx.sym.Variable('data')\n",
    "net = mx.sym.FullyConnected(data=net, name='fc1', num_hidden=128)\n",
    "net = mx.sym.Activation(data=net, name='relu1', act_type=\"relu\")\n",
    "net = mx.sym.FullyConnected(data=net, name='fc2', num_hidden=10)\n",
    "net = mx.sym.SoftmaxOutput(data=net, name='out')\n",
    "mx.viz.plot_network(net, shape={'data':(100,200)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modulelized Construction for Deep Networks\n",
    "For deep networks, such as the Google Inception, constructing layer by layer is painful given the large number of layers. For these networks, we often modularize the construction. Take the Google Inception as an example, we can first define a factory function to chain the convolution layer, batch normalization layer, and Relu activation layer together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.36.0 (20140111.2315)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"326pt\" height=\"396pt\"\n",
       " viewBox=\"0.00 0.00 326.00 396.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 392)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-392 322,-392 322,4 -4,4\"/>\n",
       "<!-- Previos Output -->\n",
       "<g id=\"node1\" class=\"node\"><title>Previos Output</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">Previos Output</text>\n",
       "</g>\n",
       "<!-- conv_None -->\n",
       "<g id=\"node2\" class=\"node\"><title>conv_None</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-168 -7.10543e-15,-168 -7.10543e-15,-110 94,-110 94,-168\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-142.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-127.8\" font-family=\"Times,serif\" font-size=\"14.00\">7x7/2, 64</text>\n",
       "</g>\n",
       "<!-- conv_None&#45;&gt;Previos Output -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>conv_None&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-99.8131C47,-86.1516 47,-71.0092 47,-58.3283\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-109.906 42.5001,-99.9062 47,-104.906 47.0001,-99.9062 47.0001,-99.9062 47.0001,-99.9062 47,-104.906 51.5001,-99.9062 47,-109.906 47,-109.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"71\" y=\"-80.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_None_gamma -->\n",
       "<g id=\"node3\" class=\"node\"><title>bn_None_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"159\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_None_gamma</text>\n",
       "</g>\n",
       "<!-- bn_None_beta -->\n",
       "<g id=\"node4\" class=\"node\"><title>bn_None_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"271\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"271\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_None_beta</text>\n",
       "</g>\n",
       "<!-- bn_None -->\n",
       "<g id=\"node5\" class=\"node\"><title>bn_None</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"206,-278 112,-278 112,-220 206,-220 206,-278\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-245.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_None&#45;&gt;conv_None -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>bn_None&#45;&gt;conv_None</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M112.664,-213.341C108.282,-209.63 103.99,-205.814 100,-202 89.1495,-191.628 78.1979,-179.216 69.0891,-168.231\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"120.606,-219.887 110.027,-216.999 116.748,-216.707 112.889,-213.527 112.889,-213.527 112.889,-213.527 116.748,-216.707 115.751,-210.054 120.606,-219.887 120.606,-219.887\"/>\n",
       "<text text-anchor=\"middle\" x=\"127.5\" y=\"-190.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x11x11</text>\n",
       "</g>\n",
       "<!-- bn_None&#45;&gt;bn_None_gamma -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>bn_None&#45;&gt;bn_None_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M159,-209.813C159,-196.152 159,-181.009 159,-168.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"159,-219.906 154.5,-209.906 159,-214.906 159,-209.906 159,-209.906 159,-209.906 159,-214.906 163.5,-209.906 159,-219.906 159,-219.906\"/>\n",
       "</g>\n",
       "<!-- bn_None&#45;&gt;bn_None_beta -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>bn_None&#45;&gt;bn_None_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.546,-212.759C212.13,-196.768 231.312,-178.27 246.186,-163.928\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"188.134,-219.906 192.209,-209.726 191.734,-216.436 195.333,-212.965 195.333,-212.965 195.333,-212.965 191.734,-216.436 198.456,-216.204 188.134,-219.906 188.134,-219.906\"/>\n",
       "</g>\n",
       "<!-- relu_None -->\n",
       "<g id=\"node6\" class=\"node\"><title>relu_None</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"206,-388 112,-388 112,-330 206,-330 206,-388\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-362.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-347.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_None&#45;&gt;bn_None -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>relu_None&#45;&gt;bn_None</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M159,-319.813C159,-306.152 159,-291.009 159,-278.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"159,-329.906 154.5,-319.906 159,-324.906 159,-319.906 159,-319.906 159,-319.906 159,-324.906 163.5,-319.906 159,-329.906 159,-329.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"186.5\" y=\"-300.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x11x11</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f55ac04c890>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Output may vary\n",
    "def ConvFactory(data, num_filter, kernel, stride=(1,1), pad=(0, 0), name=None, suffix=''):\n",
    "    conv = mx.symbol.Convolution(data=data, num_filter=num_filter, kernel=kernel, stride=stride, pad=pad, name='conv_%s%s' %(name, suffix))\n",
    "    bn = mx.symbol.BatchNorm(data=conv, name='bn_%s%s' %(name, suffix))\n",
    "    act = mx.symbol.Activation(data=bn, act_type='relu', name='relu_%s%s' %(name, suffix))\n",
    "    return act\n",
    "prev = mx.symbol.Variable(name=\"Previos Output\")\n",
    "conv_comp = ConvFactory(data=prev, num_filter=64, kernel=(7,7), stride=(2, 2))\n",
    "shape = {\"Previos Output\" : (128, 3, 28, 28)}\n",
    "mx.viz.plot_network(symbol=conv_comp, shape=shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we define a function that constructs an Inception module based on `ConvFactory`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.36.0 (20140111.2315)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"1237pt\" height=\"1292pt\"\n",
       " viewBox=\"0.00 0.00 1237.42 1292.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 1288)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-1288 1233.42,-1288 1233.42,4 -4,4\"/>\n",
       "<!-- Previos Output -->\n",
       "<g id=\"node1\" class=\"node\"><title>Previos Output</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"329.421\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"329.421\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">Previos Output</text>\n",
       "</g>\n",
       "<!-- conv_in3a_1x1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>conv_in3a_1x1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"96.4212,-878 2.42119,-878 2.42119,-820 96.4212,-820 96.4212,-878\"/>\n",
       "<text text-anchor=\"middle\" x=\"49.4212\" y=\"-852.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"49.4212\" y=\"-837.8\" font-family=\"Times,serif\" font-size=\"14.00\">1x1/1, 64</text>\n",
       "</g>\n",
       "<!-- conv_in3a_1x1&#45;&gt;Previos Output -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>conv_in3a_1x1&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M87.2667,-812.305C113.01,-784.048 142.421,-742.112 142.421,-698 142.421,-698 142.421,-698 142.421,-138 142.421,-72.9535 228.315,-46.4685 283.676,-36.1535\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"80.1306,-819.902 83.6968,-809.532 83.5537,-816.258 86.9769,-812.613 86.9769,-812.613 86.9769,-812.613 83.5537,-816.258 90.2569,-815.694 80.1306,-819.902 80.1306,-819.902\"/>\n",
       "<text text-anchor=\"middle\" x=\"166.421\" y=\"-431.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1_gamma -->\n",
       "<g id=\"node3\" class=\"node\"><title>bn_in3a_1x1_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"62.4212\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"62.4212\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_1x1_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1_beta -->\n",
       "<g id=\"node4\" class=\"node\"><title>bn_in3a_1x1_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"174.421\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"174.421\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_1x1_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1 -->\n",
       "<g id=\"node5\" class=\"node\"><title>bn_in3a_1x1</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"165.421,-1064 71.4212,-1064 71.4212,-1006 165.421,-1006 165.421,-1064\"/>\n",
       "<text text-anchor=\"middle\" x=\"118.421\" y=\"-1031.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1&#45;&gt;conv_in3a_1x1 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>bn_in3a_1x1&#45;&gt;conv_in3a_1x1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M62.252,-1012.59C38.4334,-999.842 14.3426,-980.824 6.42119,-954 -0.879655,-929.278 -3.04093,-919.978 6.42119,-896 8.94922,-889.594 12.8372,-883.554 17.2626,-878.09\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"71.3331,-1017.17 60.378,-1016.68 66.8688,-1014.91 62.4045,-1012.66 62.4045,-1012.66 62.4045,-1012.66 66.8688,-1014.91 64.4309,-1008.65 71.3331,-1017.17 71.3331,-1017.17\"/>\n",
       "<text text-anchor=\"middle\" x=\"54.9212\" y=\"-976.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1&#45;&gt;bn_in3a_1x1_gamma -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>bn_in3a_1x1&#45;&gt;bn_in3a_1x1_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M99.0935,-996.725C91.6092,-982.291 83.2077,-966.088 76.3364,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"103.854,-1005.91 95.2559,-999.1 101.552,-1001.47 99.2508,-997.029 99.2508,-997.029 99.2508,-997.029 101.552,-1001.47 103.246,-994.957 103.854,-1005.91 103.854,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_1x1&#45;&gt;bn_in3a_1x1_beta -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>bn_in3a_1x1&#45;&gt;bn_in3a_1x1_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M137.749,-996.725C145.233,-982.291 153.635,-966.088 160.506,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"132.988,-1005.91 133.597,-994.957 135.29,-1001.47 137.592,-997.029 137.592,-997.029 137.592,-997.029 135.29,-1001.47 141.586,-999.1 132.988,-1005.91 132.988,-1005.91\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_1x1 -->\n",
       "<g id=\"node6\" class=\"node\"><title>relu_in3a_1x1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"365.421,-1174 271.421,-1174 271.421,-1116 365.421,-1116 365.421,-1174\"/>\n",
       "<text text-anchor=\"middle\" x=\"318.421\" y=\"-1148.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"318.421\" y=\"-1133.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_1x1&#45;&gt;bn_in3a_1x1 -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>relu_in3a_1x1&#45;&gt;bn_in3a_1x1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M262.457,-1113.78C231.95,-1097.31 194.611,-1077.14 165.595,-1061.47\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"271.277,-1118.54 260.34,-1117.75 266.878,-1116.17 262.478,-1113.79 262.478,-1113.79 262.478,-1113.79 266.878,-1116.17 264.616,-1109.83 271.277,-1118.54 271.277,-1118.54\"/>\n",
       "<text text-anchor=\"middle\" x=\"257.921\" y=\"-1086.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_3x3_reduce -->\n",
       "<g id=\"node7\" class=\"node\"><title>conv_in3a_3x3_reduce</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"329.421,-464 235.421,-464 235.421,-406 329.421,-406 329.421,-464\"/>\n",
       "<text text-anchor=\"middle\" x=\"282.421\" y=\"-438.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"282.421\" y=\"-423.8\" font-family=\"Times,serif\" font-size=\"14.00\">1x1/1, 64</text>\n",
       "</g>\n",
       "<!-- conv_in3a_3x3_reduce&#45;&gt;Previos Output -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>conv_in3a_3x3_reduce&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M291.509,-395.739C296.794,-370.197 302.421,-335.775 302.421,-305 302.421,-305 302.421,-305 302.421,-138 302.421,-109.976 311.083,-79.0496 318.595,-57.4995\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"289.36,-405.757 287.058,-395.036 290.409,-400.868 291.458,-395.98 291.458,-395.98 291.458,-395.98 290.409,-400.868 295.858,-396.923 289.36,-405.757 289.36,-405.757\"/>\n",
       "<text text-anchor=\"middle\" x=\"326.421\" y=\"-245.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce_gamma -->\n",
       "<g id=\"node8\" class=\"node\"><title>bn_in3a_3x3_reduce_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"286.421\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"286.421\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_reduce_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce_beta -->\n",
       "<g id=\"node9\" class=\"node\"><title>bn_in3a_3x3_reduce_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"398.421\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"398.421\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_reduce_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce -->\n",
       "<g id=\"node10\" class=\"node\"><title>bn_in3a_3x3_reduce</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"333.421,-650 239.421,-650 239.421,-592 333.421,-592 333.421,-650\"/>\n",
       "<text text-anchor=\"middle\" x=\"286.421\" y=\"-617.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce&#45;&gt;conv_in3a_3x3_reduce -->\n",
       "<g id=\"edge7\" class=\"edge\"><title>bn_in3a_3x3_reduce&#45;&gt;conv_in3a_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M233.944,-584.298C231.417,-581.072 229.198,-577.638 227.421,-574 219.185,-557.135 195.55,-555.498 230.421,-482 233.521,-475.466 238.024,-469.468 243.103,-464.114\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"240.783,-591.885 230.745,-587.471 237.435,-588.172 234.087,-584.458 234.087,-584.458 234.087,-584.458 237.435,-588.172 237.43,-581.445 240.783,-591.885 240.783,-591.885\"/>\n",
       "<text text-anchor=\"middle\" x=\"254.921\" y=\"-562.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_gamma -->\n",
       "<g id=\"edge8\" class=\"edge\"><title>bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M286.421,-581.813C286.421,-568.152 286.421,-553.009 286.421,-540.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"286.421,-591.906 281.921,-581.906 286.421,-586.906 286.421,-581.906 286.421,-581.906 286.421,-581.906 286.421,-586.906 290.921,-581.906 286.421,-591.906 286.421,-591.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_beta -->\n",
       "<g id=\"edge9\" class=\"edge\"><title>bn_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M322.967,-584.759C339.551,-568.768 358.733,-550.27 373.607,-535.928\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"315.555,-591.906 319.63,-581.726 319.155,-588.436 322.754,-584.965 322.754,-584.965 322.754,-584.965 319.155,-588.436 325.878,-588.204 315.555,-591.906 315.555,-591.906\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_3x3_reduce -->\n",
       "<g id=\"node11\" class=\"node\"><title>relu_in3a_3x3_reduce</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"333.421,-802 239.421,-802 239.421,-744 333.421,-744 333.421,-802\"/>\n",
       "<text text-anchor=\"middle\" x=\"286.421\" y=\"-776.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"286.421\" y=\"-761.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce -->\n",
       "<g id=\"edge10\" class=\"edge\"><title>relu_in3a_3x3_reduce&#45;&gt;bn_in3a_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M286.421,-733.507C286.421,-707.451 286.421,-673.621 286.421,-650.087\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"286.421,-743.786 281.921,-733.786 286.421,-738.786 286.421,-733.786 286.421,-733.786 286.421,-733.786 286.421,-738.786 290.921,-733.786 286.421,-743.786 286.421,-743.786\"/>\n",
       "<text text-anchor=\"middle\" x=\"313.921\" y=\"-693.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_3x3 -->\n",
       "<g id=\"node12\" class=\"node\"><title>conv_in3a_3x3</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"333.421,-954 239.421,-954 239.421,-896 333.421,-896 333.421,-954\"/>\n",
       "<text text-anchor=\"middle\" x=\"286.421\" y=\"-928.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"286.421\" y=\"-913.8\" font-family=\"Times,serif\" font-size=\"14.00\">3x3/1, 64</text>\n",
       "</g>\n",
       "<!-- conv_in3a_3x3&#45;&gt;relu_in3a_3x3_reduce -->\n",
       "<g id=\"edge11\" class=\"edge\"><title>conv_in3a_3x3&#45;&gt;relu_in3a_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M286.421,-885.507C286.421,-859.451 286.421,-825.621 286.421,-802.087\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"286.421,-895.786 281.921,-885.786 286.421,-890.786 286.421,-885.786 286.421,-885.786 286.421,-885.786 286.421,-890.786 290.921,-885.786 286.421,-895.786 286.421,-895.786\"/>\n",
       "<text text-anchor=\"middle\" x=\"313.921\" y=\"-845.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_gamma -->\n",
       "<g id=\"node13\" class=\"node\"><title>bn_in3a_3x3_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"398.421\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"398.421\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3_beta -->\n",
       "<g id=\"node14\" class=\"node\"><title>bn_in3a_3x3_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"510.421\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"510.421\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_3x3_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3 -->\n",
       "<g id=\"node15\" class=\"node\"><title>bn_in3a_3x3</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"501.421,-1064 407.421,-1064 407.421,-1006 501.421,-1006 501.421,-1064\"/>\n",
       "<text text-anchor=\"middle\" x=\"454.421\" y=\"-1031.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3&#45;&gt;conv_in3a_3x3 -->\n",
       "<g id=\"edge12\" class=\"edge\"><title>bn_in3a_3x3&#45;&gt;conv_in3a_3x3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M398.144,-1007.42C386.747,-1001.43 375.015,-994.821 364.421,-988 348.713,-977.886 332.376,-965.271 318.737,-954.065\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"407.185,-1012.08 396.235,-1011.5 402.74,-1009.79 398.296,-1007.5 398.296,-1007.5 398.296,-1007.5 402.74,-1009.79 400.357,-1003.5 407.185,-1012.08 407.185,-1012.08\"/>\n",
       "<text text-anchor=\"middle\" x=\"391.921\" y=\"-976.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3&#45;&gt;bn_in3a_3x3_gamma -->\n",
       "<g id=\"edge13\" class=\"edge\"><title>bn_in3a_3x3&#45;&gt;bn_in3a_3x3_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M435.094,-996.725C427.609,-982.291 419.208,-966.088 412.336,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"439.854,-1005.91 431.256,-999.1 437.552,-1001.47 435.251,-997.029 435.251,-997.029 435.251,-997.029 437.552,-1001.47 439.246,-994.957 439.854,-1005.91 439.854,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_3x3&#45;&gt;bn_in3a_3x3_beta -->\n",
       "<g id=\"edge14\" class=\"edge\"><title>bn_in3a_3x3&#45;&gt;bn_in3a_3x3_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M473.749,-996.725C481.233,-982.291 489.635,-966.088 496.506,-952.836\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"468.988,-1005.91 469.597,-994.957 471.29,-1001.47 473.592,-997.029 473.592,-997.029 473.592,-997.029 471.29,-1001.47 477.586,-999.1 468.988,-1005.91 468.988,-1005.91\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_3x3 -->\n",
       "<g id=\"node16\" class=\"node\"><title>relu_in3a_3x3</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"544.421,-1174 450.421,-1174 450.421,-1116 544.421,-1116 544.421,-1174\"/>\n",
       "<text text-anchor=\"middle\" x=\"497.421\" y=\"-1148.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"497.421\" y=\"-1133.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_3x3&#45;&gt;bn_in3a_3x3 -->\n",
       "<g id=\"edge15\" class=\"edge\"><title>relu_in3a_3x3&#45;&gt;bn_in3a_3x3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M482.536,-1106.62C477.015,-1092.75 470.848,-1077.26 465.7,-1064.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"486.236,-1115.91 478.356,-1108.28 484.386,-1111.26 482.537,-1106.62 482.537,-1106.62 482.537,-1106.62 484.386,-1111.26 486.717,-1104.95 486.236,-1115.91 486.236,-1115.91\"/>\n",
       "<text text-anchor=\"middle\" x=\"506.921\" y=\"-1086.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_reduce -->\n",
       "<g id=\"node17\" class=\"node\"><title>conv_in3a_double_3x3_reduce</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"425.421,-168 331.421,-168 331.421,-110 425.421,-110 425.421,-168\"/>\n",
       "<text text-anchor=\"middle\" x=\"378.421\" y=\"-142.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"378.421\" y=\"-127.8\" font-family=\"Times,serif\" font-size=\"14.00\">1x1/1, 64</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_reduce&#45;&gt;Previos Output -->\n",
       "<g id=\"edge16\" class=\"edge\"><title>conv_in3a_double_3x3_reduce&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M361.542,-100.798C355.046,-86.4797 347.756,-70.412 341.765,-57.2072\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"365.675,-109.906 357.445,-102.659 363.609,-105.353 361.543,-100.8 361.543,-100.8 361.543,-100.8 363.609,-105.353 365.641,-98.9404 365.675,-109.906 365.675,-109.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"381.421\" y=\"-80.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce_gamma -->\n",
       "<g id=\"node18\" class=\"node\"><title>bn_in3a_double_3x3_reduce_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"490.421\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"490.421\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_reduce_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce_beta -->\n",
       "<g id=\"node19\" class=\"node\"><title>bn_in3a_double_3x3_reduce_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"602.421\" cy=\"-139\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"602.421\" y=\"-135.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_reduce_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce -->\n",
       "<g id=\"node20\" class=\"node\"><title>bn_in3a_double_3x3_reduce</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"538.421,-278 444.421,-278 444.421,-220 538.421,-220 538.421,-278\"/>\n",
       "<text text-anchor=\"middle\" x=\"491.421\" y=\"-245.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce&#45;&gt;conv_in3a_double_3x3_reduce -->\n",
       "<g id=\"edge17\" class=\"edge\"><title>bn_in3a_double_3x3_reduce&#45;&gt;conv_in3a_double_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M445.122,-213.303C440.732,-209.599 436.429,-205.796 432.421,-202 421.436,-191.596 410.289,-179.178 400.999,-168.197\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"453.073,-219.839 442.491,-216.965 449.211,-216.664 445.348,-213.488 445.348,-213.488 445.348,-213.488 449.211,-216.664 448.206,-210.012 453.073,-219.839 453.073,-219.839\"/>\n",
       "<text text-anchor=\"middle\" x=\"459.921\" y=\"-190.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_gamma -->\n",
       "<g id=\"edge18\" class=\"edge\"><title>bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M491.068,-209.813C490.941,-196.152 490.801,-181.009 490.683,-168.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"491.161,-219.906 486.569,-209.948 491.115,-214.906 491.068,-209.907 491.068,-209.907 491.068,-209.907 491.115,-214.906 495.568,-209.865 491.161,-219.906 491.161,-219.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_beta -->\n",
       "<g id=\"edge19\" class=\"edge\"><title>bn_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M527.641,-212.759C544.076,-196.768 563.088,-178.27 577.829,-163.928\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"520.295,-219.906 524.325,-209.707 523.879,-216.419 527.463,-212.933 527.463,-212.933 527.463,-212.933 523.879,-216.419 530.601,-216.158 520.295,-219.906 520.295,-219.906\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_reduce -->\n",
       "<g id=\"node21\" class=\"node\"><title>relu_in3a_double_3x3_reduce</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"553.421,-388 459.421,-388 459.421,-330 553.421,-330 553.421,-388\"/>\n",
       "<text text-anchor=\"middle\" x=\"506.421\" y=\"-362.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"506.421\" y=\"-347.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce -->\n",
       "<g id=\"edge20\" class=\"edge\"><title>relu_in3a_double_3x3_reduce&#45;&gt;bn_in3a_double_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M501.117,-319.813C499.22,-306.152 497.117,-291.009 495.356,-278.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"502.519,-329.906 496.686,-320.62 501.831,-324.954 501.144,-320.001 501.144,-320.001 501.144,-320.001 501.831,-324.954 505.601,-319.382 502.519,-329.906 502.519,-329.906\"/>\n",
       "<text text-anchor=\"middle\" x=\"527.921\" y=\"-300.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_0 -->\n",
       "<g id=\"node22\" class=\"node\"><title>conv_in3a_double_3x3_0</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"557.421,-540 463.421,-540 463.421,-482 557.421,-482 557.421,-540\"/>\n",
       "<text text-anchor=\"middle\" x=\"510.421\" y=\"-514.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"510.421\" y=\"-499.8\" font-family=\"Times,serif\" font-size=\"14.00\">3x3/1, 96</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_0&#45;&gt;relu_in3a_double_3x3_reduce -->\n",
       "<g id=\"edge21\" class=\"edge\"><title>conv_in3a_double_3x3_0&#45;&gt;relu_in3a_double_3x3_reduce</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M509.395,-471.507C508.7,-445.451 507.798,-411.621 507.17,-388.087\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"509.669,-481.786 504.904,-471.91 509.535,-476.788 509.402,-471.79 509.402,-471.79 509.402,-471.79 509.535,-476.788 513.901,-471.67 509.669,-481.786 509.669,-481.786\"/>\n",
       "<text text-anchor=\"middle\" x=\"536.921\" y=\"-431.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0_gamma -->\n",
       "<g id=\"node23\" class=\"node\"><title>bn_in3a_double_3x3_0_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"622.421\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"622.421\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_0_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0_beta -->\n",
       "<g id=\"node24\" class=\"node\"><title>bn_in3a_double_3x3_0_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"734.421\" cy=\"-511\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"734.421\" y=\"-507.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_0_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0 -->\n",
       "<g id=\"node25\" class=\"node\"><title>bn_in3a_double_3x3_0</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"669.421,-650 575.421,-650 575.421,-592 669.421,-592 669.421,-650\"/>\n",
       "<text text-anchor=\"middle\" x=\"622.421\" y=\"-617.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0&#45;&gt;conv_in3a_double_3x3_0 -->\n",
       "<g id=\"edge22\" class=\"edge\"><title>bn_in3a_double_3x3_0&#45;&gt;conv_in3a_double_3x3_0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M576.086,-585.341C571.703,-581.63 567.411,-577.814 563.421,-574 552.571,-563.628 541.619,-551.216 532.51,-540.231\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"584.028,-591.887 573.449,-588.999 580.169,-588.707 576.311,-585.527 576.311,-585.527 576.311,-585.527 580.169,-588.707 579.173,-582.054 584.028,-591.887 584.028,-591.887\"/>\n",
       "<text text-anchor=\"middle\" x=\"590.921\" y=\"-562.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_gamma -->\n",
       "<g id=\"edge23\" class=\"edge\"><title>bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M622.421,-581.813C622.421,-568.152 622.421,-553.009 622.421,-540.328\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"622.421,-591.906 617.921,-581.906 622.421,-586.906 622.421,-581.906 622.421,-581.906 622.421,-581.906 622.421,-586.906 626.921,-581.906 622.421,-591.906 622.421,-591.906\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_beta -->\n",
       "<g id=\"edge24\" class=\"edge\"><title>bn_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M658.967,-584.759C675.551,-568.768 694.733,-550.27 709.607,-535.928\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"651.555,-591.906 655.63,-581.726 655.155,-588.436 658.754,-584.965 658.754,-584.965 658.754,-584.965 655.155,-588.436 661.878,-588.204 651.555,-591.906 651.555,-591.906\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_0 -->\n",
       "<g id=\"node26\" class=\"node\"><title>relu_in3a_double_3x3_0</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"669.421,-802 575.421,-802 575.421,-744 669.421,-744 669.421,-802\"/>\n",
       "<text text-anchor=\"middle\" x=\"622.421\" y=\"-776.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"622.421\" y=\"-761.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0 -->\n",
       "<g id=\"edge25\" class=\"edge\"><title>relu_in3a_double_3x3_0&#45;&gt;bn_in3a_double_3x3_0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M622.421,-733.507C622.421,-707.451 622.421,-673.621 622.421,-650.087\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"622.421,-743.786 617.921,-733.786 622.421,-738.786 622.421,-733.786 622.421,-733.786 622.421,-733.786 622.421,-738.786 626.921,-733.786 622.421,-743.786 622.421,-743.786\"/>\n",
       "<text text-anchor=\"middle\" x=\"649.921\" y=\"-693.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_1 -->\n",
       "<g id=\"node27\" class=\"node\"><title>conv_in3a_double_3x3_1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"669.421,-954 575.421,-954 575.421,-896 669.421,-896 669.421,-954\"/>\n",
       "<text text-anchor=\"middle\" x=\"622.421\" y=\"-928.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"622.421\" y=\"-913.8\" font-family=\"Times,serif\" font-size=\"14.00\">3x3/1, 96</text>\n",
       "</g>\n",
       "<!-- conv_in3a_double_3x3_1&#45;&gt;relu_in3a_double_3x3_0 -->\n",
       "<g id=\"edge26\" class=\"edge\"><title>conv_in3a_double_3x3_1&#45;&gt;relu_in3a_double_3x3_0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M622.421,-885.507C622.421,-859.451 622.421,-825.621 622.421,-802.087\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"622.421,-895.786 617.921,-885.786 622.421,-890.786 622.421,-885.786 622.421,-885.786 622.421,-885.786 622.421,-890.786 626.921,-885.786 622.421,-895.786 622.421,-895.786\"/>\n",
       "<text text-anchor=\"middle\" x=\"649.921\" y=\"-845.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1_gamma -->\n",
       "<g id=\"node28\" class=\"node\"><title>bn_in3a_double_3x3_1_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"734.421\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"734.421\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_1_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1_beta -->\n",
       "<g id=\"node29\" class=\"node\"><title>bn_in3a_double_3x3_1_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"846.421\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"846.421\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_double_3x3_1_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1 -->\n",
       "<g id=\"node30\" class=\"node\"><title>bn_in3a_double_3x3_1</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"735.421,-1064 641.421,-1064 641.421,-1006 735.421,-1006 735.421,-1064\"/>\n",
       "<text text-anchor=\"middle\" x=\"688.421\" y=\"-1031.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1&#45;&gt;conv_in3a_double_3x3_1 -->\n",
       "<g id=\"edge27\" class=\"edge\"><title>bn_in3a_double_3x3_1&#45;&gt;conv_in3a_double_3x3_1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M656.162,-997.874C653.763,-994.621 651.479,-991.305 649.421,-988 642.843,-977.436 637.083,-964.991 632.592,-954.033\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"662.304,-1005.79 652.619,-1000.65 659.239,-1001.84 656.174,-997.889 656.174,-997.889 656.174,-997.889 659.239,-1001.84 659.73,-995.131 662.304,-1005.79 662.304,-1005.79\"/>\n",
       "<text text-anchor=\"middle\" x=\"676.921\" y=\"-976.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_gamma -->\n",
       "<g id=\"edge28\" class=\"edge\"><title>bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M704.441,-996.388C710.495,-982.174 717.262,-966.286 722.833,-953.207\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"700.387,-1005.91 700.166,-994.943 702.346,-1001.31 704.306,-996.706 704.306,-996.706 704.306,-996.706 702.346,-1001.31 708.446,-998.469 700.387,-1005.91 700.387,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_beta -->\n",
       "<g id=\"edge29\" class=\"edge\"><title>bn_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M737.931,-1000.16C763.319,-982.804 793.452,-962.207 815.51,-947.129\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"729.521,-1005.91 735.238,-996.548 733.649,-1003.08 737.777,-1000.26 737.777,-1000.26 737.777,-1000.26 733.649,-1003.08 740.316,-1003.98 729.521,-1005.91 729.521,-1005.91\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_1 -->\n",
       "<g id=\"node31\" class=\"node\"><title>relu_in3a_double_3x3_1</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"669.421,-1174 575.421,-1174 575.421,-1116 669.421,-1116 669.421,-1174\"/>\n",
       "<text text-anchor=\"middle\" x=\"622.421\" y=\"-1148.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"622.421\" y=\"-1133.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1 -->\n",
       "<g id=\"edge30\" class=\"edge\"><title>relu_in3a_double_3x3_1&#45;&gt;bn_in3a_double_3x3_1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M645.023,-1107.01C653.561,-1093.04 663.132,-1077.38 671.109,-1064.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"639.59,-1115.91 640.964,-1105.03 642.197,-1111.64 644.804,-1107.37 644.804,-1107.37 644.804,-1107.37 642.197,-1111.64 648.644,-1109.72 639.59,-1115.91 639.59,-1115.91\"/>\n",
       "<text text-anchor=\"middle\" x=\"686.921\" y=\"-1086.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- avg_pool_in3a_pool -->\n",
       "<g id=\"node32\" class=\"node\"><title>avg_pool_in3a_pool</title>\n",
       "<polygon fill=\"#80b1d3\" stroke=\"black\" points=\"949.421,-726 855.421,-726 855.421,-668 949.421,-668 949.421,-726\"/>\n",
       "<text text-anchor=\"middle\" x=\"902.421\" y=\"-700.8\" font-family=\"Times,serif\" font-size=\"14.00\">Pooling</text>\n",
       "<text text-anchor=\"middle\" x=\"902.421\" y=\"-685.8\" font-family=\"Times,serif\" font-size=\"14.00\">avg, 3x3/1</text>\n",
       "</g>\n",
       "<!-- avg_pool_in3a_pool&#45;&gt;Previos Output -->\n",
       "<g id=\"edge31\" class=\"edge\"><title>avg_pool_in3a_pool&#45;&gt;Previos Output</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M868.352,-660.02C848.929,-635.705 828.421,-601.795 828.421,-567 828.421,-567 828.421,-567 828.421,-138 828.421,-46.1491 501.375,-32.2341 376.816,-30.2652\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"874.866,-667.885 865.022,-663.054 871.677,-664.034 868.488,-660.183 868.488,-660.183 868.488,-660.183 871.677,-664.034 871.953,-657.313 874.866,-667.885 874.866,-667.885\"/>\n",
       "<text text-anchor=\"middle\" x=\"852.421\" y=\"-355.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- conv_in3a_proj -->\n",
       "<g id=\"node33\" class=\"node\"><title>conv_in3a_proj</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"1005.42,-954 911.421,-954 911.421,-896 1005.42,-896 1005.42,-954\"/>\n",
       "<text text-anchor=\"middle\" x=\"958.421\" y=\"-928.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"958.421\" y=\"-913.8\" font-family=\"Times,serif\" font-size=\"14.00\">1x1/1, 32</text>\n",
       "</g>\n",
       "<!-- conv_in3a_proj&#45;&gt;avg_pool_in3a_pool -->\n",
       "<g id=\"edge32\" class=\"edge\"><title>conv_in3a_proj&#45;&gt;avg_pool_in3a_pool</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M949.034,-886.117C937.844,-840.957 919.404,-766.537 909.4,-726.165\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"951.442,-895.834 944.669,-887.21 950.239,-890.981 949.037,-886.127 949.037,-886.127 949.037,-886.127 950.239,-890.981 953.405,-885.045 951.442,-895.834 951.442,-895.834\"/>\n",
       "<text text-anchor=\"middle\" x=\"969.421\" y=\"-845.3\" font-family=\"Times,serif\" font-size=\"14.00\">3x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj_gamma -->\n",
       "<g id=\"node34\" class=\"node\"><title>bn_in3a_proj_gamma</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1070.42\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1070.42\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_proj_gamma</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj_beta -->\n",
       "<g id=\"node35\" class=\"node\"><title>bn_in3a_proj_beta</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"1182.42\" cy=\"-925\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"1182.42\" y=\"-921.3\" font-family=\"Times,serif\" font-size=\"14.00\">bn_in3a_proj_beta</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj -->\n",
       "<g id=\"node36\" class=\"node\"><title>bn_in3a_proj</title>\n",
       "<polygon fill=\"#bebada\" stroke=\"black\" points=\"1071.42,-1064 977.421,-1064 977.421,-1006 1071.42,-1006 1071.42,-1064\"/>\n",
       "<text text-anchor=\"middle\" x=\"1024.42\" y=\"-1031.3\" font-family=\"Times,serif\" font-size=\"14.00\">BatchNorm</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj&#45;&gt;conv_in3a_proj -->\n",
       "<g id=\"edge33\" class=\"edge\"><title>bn_in3a_proj&#45;&gt;conv_in3a_proj</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M992.162,-997.874C989.763,-994.621 987.479,-991.305 985.421,-988 978.843,-977.436 973.083,-964.991 968.592,-954.033\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"998.304,-1005.79 988.619,-1000.65 995.239,-1001.84 992.174,-997.889 992.174,-997.889 992.174,-997.889 995.239,-1001.84 995.73,-995.131 998.304,-1005.79 998.304,-1005.79\"/>\n",
       "<text text-anchor=\"middle\" x=\"1012.92\" y=\"-976.3\" font-family=\"Times,serif\" font-size=\"14.00\">32x28x28</text>\n",
       "</g>\n",
       "<!-- bn_in3a_proj&#45;&gt;bn_in3a_proj_gamma -->\n",
       "<g id=\"edge34\" class=\"edge\"><title>bn_in3a_proj&#45;&gt;bn_in3a_proj_gamma</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1040.44,-996.388C1046.5,-982.174 1053.26,-966.286 1058.83,-953.207\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1036.39,-1005.91 1036.17,-994.943 1038.35,-1001.31 1040.31,-996.706 1040.31,-996.706 1040.31,-996.706 1038.35,-1001.31 1044.45,-998.469 1036.39,-1005.91 1036.39,-1005.91\"/>\n",
       "</g>\n",
       "<!-- bn_in3a_proj&#45;&gt;bn_in3a_proj_beta -->\n",
       "<g id=\"edge35\" class=\"edge\"><title>bn_in3a_proj&#45;&gt;bn_in3a_proj_beta</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1073.93,-1000.16C1099.32,-982.804 1129.45,-962.207 1151.51,-947.129\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1065.52,-1005.91 1071.24,-996.548 1069.65,-1003.08 1073.78,-1000.26 1073.78,-1000.26 1073.78,-1000.26 1069.65,-1003.08 1076.32,-1003.98 1065.52,-1005.91 1065.52,-1005.91\"/>\n",
       "</g>\n",
       "<!-- relu_in3a_proj -->\n",
       "<g id=\"node37\" class=\"node\"><title>relu_in3a_proj</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"844.421,-1174 750.421,-1174 750.421,-1116 844.421,-1116 844.421,-1174\"/>\n",
       "<text text-anchor=\"middle\" x=\"797.421\" y=\"-1148.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"797.421\" y=\"-1133.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- relu_in3a_proj&#45;&gt;bn_in3a_proj -->\n",
       "<g id=\"edge36\" class=\"edge\"><title>relu_in3a_proj&#45;&gt;bn_in3a_proj</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M853.834,-1117.16C891.758,-1099.12 941.178,-1075.6 977.057,-1058.53\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"844.456,-1121.62 851.552,-1113.26 848.971,-1119.47 853.486,-1117.33 853.486,-1117.33 853.486,-1117.33 848.971,-1119.47 855.419,-1121.39 844.456,-1121.62 844.456,-1121.62\"/>\n",
       "<text text-anchor=\"middle\" x=\"951.921\" y=\"-1086.3\" font-family=\"Times,serif\" font-size=\"14.00\">32x28x28</text>\n",
       "</g>\n",
       "<!-- ch_concat_in3a_chconcat -->\n",
       "<g id=\"node38\" class=\"node\"><title>ch_concat_in3a_chconcat</title>\n",
       "<polygon fill=\"#fdb462\" stroke=\"black\" points=\"606.421,-1284 512.421,-1284 512.421,-1226 606.421,-1226 606.421,-1284\"/>\n",
       "<text text-anchor=\"middle\" x=\"559.421\" y=\"-1251.3\" font-family=\"Times,serif\" font-size=\"14.00\">Concat</text>\n",
       "</g>\n",
       "<!-- ch_concat_in3a_chconcat&#45;&gt;relu_in3a_1x1 -->\n",
       "<g id=\"edge37\" class=\"edge\"><title>ch_concat_in3a_chconcat&#45;&gt;relu_in3a_1x1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M503.177,-1228.8C461.448,-1210.09 405.188,-1184.88 365.753,-1167.21\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"512.382,-1232.92 501.416,-1232.94 507.819,-1230.88 503.256,-1228.83 503.256,-1228.83 503.256,-1228.83 507.819,-1230.88 505.097,-1224.72 512.382,-1232.92 512.382,-1232.92\"/>\n",
       "<text text-anchor=\"middle\" x=\"480.921\" y=\"-1196.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- ch_concat_in3a_chconcat&#45;&gt;relu_in3a_3x3 -->\n",
       "<g id=\"edge38\" class=\"edge\"><title>ch_concat_in3a_chconcat&#45;&gt;relu_in3a_3x3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M538.189,-1217.01C530.169,-1203.04 521.178,-1187.38 513.684,-1174.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"543.293,-1225.91 534.412,-1219.47 540.804,-1221.57 538.315,-1217.23 538.315,-1217.23 538.315,-1217.23 540.804,-1221.57 542.217,-1214.99 543.293,-1225.91 543.293,-1225.91\"/>\n",
       "<text text-anchor=\"middle\" x=\"559.921\" y=\"-1196.3\" font-family=\"Times,serif\" font-size=\"14.00\">64x28x28</text>\n",
       "</g>\n",
       "<!-- ch_concat_in3a_chconcat&#45;&gt;relu_in3a_double_3x3_1 -->\n",
       "<g id=\"edge39\" class=\"edge\"><title>ch_concat_in3a_chconcat&#45;&gt;relu_in3a_double_3x3_1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M582.869,-1217.25C584.771,-1214.13 586.643,-1211.02 588.421,-1208 594.836,-1197.1 601.593,-1184.94 607.378,-1174.29\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"577.486,-1225.95 578.92,-1215.08 580.116,-1221.7 582.747,-1217.44 582.747,-1217.44 582.747,-1217.44 580.116,-1221.7 586.574,-1219.81 577.486,-1225.95 577.486,-1225.95\"/>\n",
       "<text text-anchor=\"middle\" x=\"624.921\" y=\"-1196.3\" font-family=\"Times,serif\" font-size=\"14.00\">96x28x28</text>\n",
       "</g>\n",
       "<!-- ch_concat_in3a_chconcat&#45;&gt;relu_in3a_proj -->\n",
       "<g id=\"edge40\" class=\"edge\"><title>ch_concat_in3a_chconcat&#45;&gt;relu_in3a_proj</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M615.739,-1228.44C656.66,-1209.87 711.421,-1185.03 750.091,-1167.48\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"606.442,-1232.66 613.689,-1224.43 610.995,-1230.6 615.549,-1228.53 615.549,-1228.53 615.549,-1228.53 610.995,-1230.6 617.408,-1232.63 606.442,-1232.66 606.442,-1232.66\"/>\n",
       "<text text-anchor=\"middle\" x=\"720.921\" y=\"-1196.3\" font-family=\"Times,serif\" font-size=\"14.00\">32x28x28</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f55803e2490>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Output may vary\n",
    "def InceptionFactoryA(data, num_1x1, num_3x3red, num_3x3, num_d3x3red, num_d3x3, pool, proj, name):\n",
    "    # 1x1\n",
    "    c1x1 = ConvFactory(data=data, num_filter=num_1x1, kernel=(1, 1), name=('%s_1x1' % name))\n",
    "    # 3x3 reduce + 3x3\n",
    "    c3x3r = ConvFactory(data=data, num_filter=num_3x3red, kernel=(1, 1), name=('%s_3x3' % name), suffix='_reduce')\n",
    "    c3x3 = ConvFactory(data=c3x3r, num_filter=num_3x3, kernel=(3, 3), pad=(1, 1), name=('%s_3x3' % name))\n",
    "    # double 3x3 reduce + double 3x3\n",
    "    cd3x3r = ConvFactory(data=data, num_filter=num_d3x3red, kernel=(1, 1), name=('%s_double_3x3' % name), suffix='_reduce')\n",
    "    cd3x3 = ConvFactory(data=cd3x3r, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), name=('%s_double_3x3_0' % name))\n",
    "    cd3x3 = ConvFactory(data=cd3x3, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), name=('%s_double_3x3_1' % name))\n",
    "    # pool + proj\n",
    "    pooling = mx.symbol.Pooling(data=data, kernel=(3, 3), stride=(1, 1), pad=(1, 1), pool_type=pool, name=('%s_pool_%s_pool' % (pool, name)))\n",
    "    cproj = ConvFactory(data=pooling, num_filter=proj, kernel=(1, 1), name=('%s_proj' %  name))\n",
    "    # concat\n",
    "    concat = mx.symbol.Concat(*[c1x1, c3x3, cd3x3, cproj], name='ch_concat_%s_chconcat' % name)\n",
    "    return concat\n",
    "prev = mx.symbol.Variable(name=\"Previos Output\")\n",
    "in3a = InceptionFactoryA(prev, 64, 64, 64, 64, 96, \"avg\", 32, name=\"in3a\")\n",
    "mx.viz.plot_network(symbol=in3a, shape=shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we can obtain the whole network by chaining multiple inception modulas. A complete example is available at [mxnet/example/image-classification/symbol_inception-bn.py](https://github.com/dmlc/mxnet/blob/master/example/image-classification/symbol_inception-bn.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group Multiple Symbols\n",
    "\n",
    "To construct neural networks with multiple loss layers, we can use `mxnet.sym.Group` to group multiple symbols together. The following example group two outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['softmax_output', 'regression_output']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = mx.sym.Variable('data')\n",
    "fc1 = mx.sym.FullyConnected(data=net, name='fc1', num_hidden=128)\n",
    "net = mx.sym.Activation(data=fc1, name='relu1', act_type=\"relu\")\n",
    "out1 = mx.sym.SoftmaxOutput(data=net, name='softmax')\n",
    "out2 = mx.sym.LinearRegressionOutput(data=net, name='regression')\n",
    "group = mx.sym.Group([out1, out2])\n",
    "group.list_outputs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relations to NDArray\n",
    "\n",
    "As can be seen now, both Symbol and NDArray provide multi-dimensional array operations, such as `c=a+b` in MXNet. Sometimes users are confused which way to use. We briefly clarify the difference here, more detailed explanation are available [here](http://mxnet.readthedocs.io/en/latest/system/program_model.html). \n",
    "\n",
    "The `NDArray` provides an imperative programming alike interface, in which the computations are evaluated sentence by sentence. While `Symbol` is closer to declarative programming, in which we first declare the computation, and then evaluate with data. Examples in this category include regular expression and SQL.\n",
    "\n",
    "The pros for `NDArray`:\n",
    "- straightforward\n",
    "- easy to work with other language features (for loop, if-else condition, ..) and libraries (numpy, ..)\n",
    "- easy to step-by-step debug\n",
    "\n",
    "The pros for `Symbol`:\n",
    "- provides almost all functionalities of NDArray, such as +, \\*, sin, and reshape \n",
    "- provides a large number of neural network related operators such as Convolution, Activation, and BatchNorm\n",
    "- provides automatic differentiation \n",
    "- easy to construct and manipulate complex computations such as deep neural networks\n",
    "- easy to save, load, and visualization\n",
    "- easy for the backend to optimize the computation and memory usage\n",
    "\n",
    "We will show on the [mixed programming tutorial](./mixed.ipynb) how these two interfaces can be used together to develop a complete training program. This tutorial will focus on the usage of Symbol. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Symbol Manipulation *\n",
    "\n",
    "One important difference of `Symbol` comparing to `NDArray` is that, we first declare the computation, and then bind with data to run. \n",
    "\n",
    "In this section we introduce the functions to manipulate a symbol directly. But note that, most of them are wrapped nicely by the [`mx.module`](./module.ipynb). One can skip this section safely.  \n",
    "\n",
    "### Shape Inference\n",
    "For each symbol, we can query its inputs (or arguments) and outputs. We can also inference the output shape by given the input shape, which facilitates memory allocation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input': {'a': (2L, 3L), 'b': (2L, 3L)},\n",
       " 'output': {'_plus0_output': (2L, 3L)}}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arg_name = c.list_arguments()  # get the names of the inputs\n",
    "out_name = c.list_outputs()    # get the names of the outputs\n",
    "arg_shape, out_shape, _ = c.infer_shape(a=(2,3), b=(2,3))  \n",
    "{'input' : dict(zip(arg_name, arg_shape)), \n",
    " 'output' : dict(zip(out_name, out_shape))}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bind with Data and Evaluate\n",
    "The symbol `c` we constructed declares what computation should be run. To evaluate it, we need to feed arguments, namely free variables, with data first. We can do it by using the `bind` method, which accepts device context and a `dict` mapping free variable names to `NDArray`s as arguments and returns an executor. The executor provides method `forward` for evaluation and attribute `outputs` to get all results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"test_ndarray_element_value\"\"\"\n",
    "import numpy as np\n",
    "\n",
    "def test_val(nd_array, val):\n",
    "    for num in np.nditer(nd_array):\n",
    "        assert round(num, 7) == round(val, 7), \"NDArray element value incorrect.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of outputs = 1\n",
      "the first output = \n",
      "[[ 2.  2.  2.]\n",
      " [ 2.  2.  2.]]\n"
     ]
    }
   ],
   "source": [
    "ex = c.bind(ctx=mx.cpu(), args={'a' : mx.nd.ones([2,3]), \n",
    "                                'b' : mx.nd.ones([2,3])})\n",
    "ex.forward()\n",
    "test_val(ex.outputs[0].asnumpy(), 2)\n",
    "print 'number of outputs = %d\\nthe first output = \\n%s' % (\n",
    "           len(ex.outputs), ex.outputs[0].asnumpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can evaluate the same symbol on GPU with different data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5.,  5.,  5.,  5.],\n",
       "       [ 5.,  5.,  5.,  5.],\n",
       "       [ 5.,  5.,  5.,  5.]], dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex_gpu = c.bind(ctx=mx.gpu(), args={'a' : mx.nd.ones([3,4])*2,\n",
    "                                    'b' : mx.nd.ones([3,4])*3})\n",
    "ex_gpu.forward()\n",
    "ex_gpu.outputs[0].asnumpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Load and Save\n",
    "\n",
    "Similar to NDArray, we can either serialize a `Symbol` object by using `pickle`, or use `save` and `load` directly. Different to the binary format chosen by `NDArray`, `Symbol` uses the more readable json format for serialization. The `tojson` method returns the json string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"nodes\": [\n",
      "    {\n",
      "      \"op\": \"null\", \n",
      "      \"param\": {}, \n",
      "      \"name\": \"a\", \n",
      "      \"inputs\": [], \n",
      "      \"backward_source_id\": -1\n",
      "    }, \n",
      "    {\n",
      "      \"op\": \"null\", \n",
      "      \"param\": {}, \n",
      "      \"name\": \"b\", \n",
      "      \"inputs\": [], \n",
      "      \"backward_source_id\": -1\n",
      "    }, \n",
      "    {\n",
      "      \"op\": \"_Plus\", \n",
      "      \"param\": {}, \n",
      "      \"name\": \"_plus0\", \n",
      "      \"inputs\": [[0, 0], [1, 0]], \n",
      "      \"backward_source_id\": -1\n",
      "    }\n",
      "  ], \n",
      "  \"arg_nodes\": [0, 1], \n",
      "  \"heads\": [[2, 0]]\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(c.tojson())\n",
    "c.save('symbol-c.json')\n",
    "c2 = mx.symbol.load('symbol-c.json')\n",
    "c.tojson() == c2.tojson()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Auto Differentiation\n",
    "\n",
    "Auto differentiation is one of the most important feature Symbol differs to NDArray. We can use the method `grad` to obtain the symbol for calculating the gradient. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['_plus0_0_grad', 'a', 'b']\n",
      "['_plus0_backward_lhs']\n",
      "[[ 5.  5.  5.  5.]\n",
      " [ 5.  5.  5.  5.]\n",
      " [ 5.  5.  5.  5.]]\n"
     ]
    }
   ],
   "source": [
    "x = c.grad(wrt=('a'))\n",
    "print x.list_arguments()\n",
    "print x.list_outputs()\n",
    "\n",
    "ex2 = x.bind(ctx=mx.cpu(), args={c.name+'_0_grad': mx.nd.ones([3,4])*5, \n",
    "                                 'a': mx.nd.ones([3,4])*2,\n",
    "                                 'b' : mx.nd.ones([3,4])*3})\n",
    "ex2.forward()\n",
    "test_val(ex2.outputs[0].asnumpy(), 5)\n",
    "print ex2.outputs[0].asnumpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customized Symbol *\n",
    "\n",
    "Most operators such as `mx.sym.Convolution` and `mx.sym.Reshape` are implemented in C++ for better performance. MXNet also allows users to write new operators using any frontend language such as Python. It often makes the developing and debugging much easier. \n",
    "\n",
    "To implement an operator in Python, we just need to define the two computation methods `forward` and `backward` with several methods for querying the properties, such as `list_arguments` and `infer_shape`. \n",
    "\n",
    "`NDArray` is the default type of arguments in both `forward` and `backward`. Therefore we often also implement the computation with `NDArray` operations. To show the flexibility of MXNet, however, we will demonstrate an implementation of the `softmax` layer using NumPy. Though a NumPy based operator can be only run on CPU and also lose some optimizations which can be applied on NDArray, it enjoys the rich functionalities provided by NumPy.\n",
    "\n",
    "We first create a subclass of `mx.operator.CustomOp` and then define `forward` and `backward`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Softmax(mx.operator.CustomOp):\n",
    "    def forward(self, is_train, req, in_data, out_data, aux):\n",
    "        x = in_data[0].asnumpy()\n",
    "        y = np.exp(x - x.max(axis=1).reshape((x.shape[0], 1)))\n",
    "        y /= y.sum(axis=1).reshape((x.shape[0], 1))\n",
    "        self.assign(out_data[0], req[0], mx.nd.array(y))\n",
    "\n",
    "    def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n",
    "        l = in_data[1].asnumpy().ravel().astype(np.int)\n",
    "        y = out_data[0].asnumpy()\n",
    "        y[np.arange(l.shape[0]), l] -= 1.0\n",
    "        self.assign(in_grad[0], req[0], mx.nd.array(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Here we use `asnumpy` to convert the `NDArray` inputs into `numpy.ndarray`. Then using `CustomOp.assign` to assign the results back to `mxnet.NDArray` based on the value of req, which could be \"over write\" or \"add to\". \n",
    "\n",
    "Next we create a subclass of `mx.operator.CustomOpProp` for querying the properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# register this operator into MXNet by name \"softmax\"\n",
    "@mx.operator.register(\"softmax\")\n",
    "class SoftmaxProp(mx.operator.CustomOpProp):\n",
    "    def __init__(self):\n",
    "        # softmax is a loss layer so we don’t need gradient input\n",
    "        # from layers above. \n",
    "        super(SoftmaxProp, self).__init__(need_top_grad=False)\n",
    "    \n",
    "    def list_arguments(self):\n",
    "        return ['data', 'label']\n",
    "\n",
    "    def list_outputs(self):\n",
    "        return ['output']\n",
    "\n",
    "    def infer_shape(self, in_shape):\n",
    "        data_shape = in_shape[0]\n",
    "        label_shape = (in_shape[0][0],)\n",
    "        output_shape = in_shape[0]\n",
    "        return [data_shape, label_shape], [output_shape], []\n",
    "\n",
    "    def create_operator(self, ctx, shapes, dtypes):\n",
    "        return Softmax()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can use `mx.sym.Custom` with the register name to use this operator\n",
    "```python\n",
    "net = mx.symbol.Custom(data=prev_input, op_type='softmax')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Usages *\n",
    "\n",
    "### Type Cast\n",
    "MXNet uses 32-bit float in default. Sometimes we want to use a lower precision data type for better accuracy-performance trade-off. For example, The Nvidia Tesla Pascal GPUs (e.g. P100) have improved 16-bit float performance, while GTX Pascal GPUs (e.g. GTX 1080) are fast on 8-bit integers. \n",
    "\n",
    "We can use the `mx.sym.Cast` operator to convert the data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input': [<type 'numpy.float32'>], 'output': [<type 'numpy.float16'>]}\n",
      "{'input': [<type 'numpy.int32'>], 'output': [<type 'numpy.uint8'>]}\n"
     ]
    }
   ],
   "source": [
    "a = mx.sym.Variable('data')\n",
    "b = mx.sym.Cast(data=a, dtype='float16')\n",
    "arg, out, _ = b.infer_type(data='float32')\n",
    "assert out[0] is np.float16, \"Type cast failed.\"\n",
    "print({'input':arg, 'output':out})\n",
    "\n",
    "c = mx.sym.Cast(data=a, dtype='uint8')\n",
    "arg, out, _ = c.infer_type(data='int32')\n",
    "assert out[0] is np.uint8, \"Type cast failed.\"\n",
    "print({'input':arg, 'output':out})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variable Sharing\n",
    "\n",
    "Sometimes we want to share the contents between several symbols. This can be simply done by bind these symbols with the same array. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6.,  6.,  6.],\n",
       "       [ 6.,  6.,  6.]], dtype=float32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = mx.sym.Variable('a')\n",
    "b = mx.sym.Variable('b')\n",
    "c = mx.sym.Variable('c')\n",
    "d = a + b * c\n",
    "\n",
    "data = mx.nd.ones((2,3))*2\n",
    "ex = d.bind(ctx=mx.cpu(), args={'a':data, 'b':data, 'c':data})\n",
    "ex.forward()\n",
    "test_val(ex.outputs[0].asnumpy(), 6)\n",
    "ex.outputs[0].asnumpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Symbol Attributes\n",
    "Attributes can be attached to symbols to provide additional information.\n",
    "For example, we can pass the device information for model parallelism, or layer specific learning rate. \n",
    "\n",
    "The attributes are string key-value pairs, which can be retrieved via `attr(key)` or `list_attr()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('angry', {'mood': 'so so'})"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = mx.sym.Variable('data', attr={'mood': 'angry'})\n",
    "op   = mx.sym.Convolution(data=data, name='conv', kernel=(1, 1),\n",
    "                          num_filter=1, attr={'mood': 'so so'})\n",
    "assert data.attr('mood') == 'angry', \"Symbol attribute creation failed.\"\n",
    "(data.attr('mood'), op.list_attr())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can use `mx.sym.Custom` with the register name to use this operator\n",
    "```python\n",
    "net = mx.symbol.Custom(data=prev_input, op_type='softmax')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also retrieve all the attributes associated with that symbol and its descendants via `list_attr(recursive=True)`. Note that the returned attribute names are added with the symbol name prefix to avoid naming conflicts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'conv_bias_mood': 'so so',\n",
       " 'conv_mood': 'so so',\n",
       " 'conv_weight_mood': 'so so',\n",
       " 'data_mood': 'angry'}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attr_dict = op.list_attr(recursive=True)\n",
    "assert attr_dict['conv_bias_mood'] == 'so so' and attr_dict['conv_mood'] == 'so so' and attr_dict['conv_weight_mood'] == 'so so' and attr_dict['data_mood'] == 'angry', \"Symbol attribute creation failed.\"\n",
    "attr_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `mood` attribute we set for the Convolution operator is copied to both parameters `conv_weight` and `conv_bias`. This copy is applied in default. We can also use `AttrScope` to add attributes to all symbols in the scope."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('great', 'great', '0.0')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = mx.symbol.Variable('data')\n",
    "with mx.AttrScope(group='4', data='great'):\n",
    "    fc1 = mx.symbol.Activation(data, act_type='relu')\n",
    "    with mx.AttrScope(init_bias='0.0'):\n",
    "        fc2 = mx.symbol.FullyConnected(fc1, num_hidden=10, name='fc2')\n",
    "assert fc1.attr('data') == 'great' and fc2.attr('data') == 'great' and fc2.attr('init_bias') == '0.0', \"AttrScope function fail.\"\n",
    "(fc1.attr('data'), fc2.attr('data'), fc2.attr('init_bias'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can explicitly create the parameter to stop the default copy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'conv_bias_mood': 'so so', 'conv_mood': 'so so', 'crazy_weight_size': '5'}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weight = mx.sym.Variable('crazy_weight', attr={'size': '5'})\n",
    "op = mx.sym.Convolution(data=data, weight=weight, name='conv', kernel=(1, 1),\n",
    "                        num_filter=1, attr={'mood': 'so so'})\n",
    "attr_dict = op.list_attr(recursive=True)\n",
    "assert attr_dict['conv_bias_mood'] == 'so so' and attr_dict['conv_mood'] == 'so so' and attr_dict['crazy_weight_size'] == '5', \"Symbol attribute creation failed.\"\n",
    "attr_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Readings\n",
    "\n",
    "- [Use torch operators in MXNet](http://mxnet.dmlc.ml/en/latest/how_to/torch.html)\n",
    "- [Use Caffe operators in MXNet](http://dmlc.ml/mxnet/2016/07/29/use-caffe-operator-in-mxnet.html)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
